Imaging device

ABSTRACT

Provided is an imaging device, including: an image acquisition unit for acquiring a plurality of images having different focal positions; an orientation acquisition unit for acquiring orientation information of the image acquisition unit; and an image processing unit for generating, from the plurality of images, an image having a depth of field which is enlarged compared to a depth of field of one of the plurality of images, in which a focal position setting value by which the focal position is determined is provided, and the focal position setting value is corrected based on the orientation information.

TECHNICAL FIELD

The present invention relates to an imaging technique of acquiring animage having an enlarged depth of field.

BACKGROUND ART

Imaging devices, such as a camera mounted in a mobile terminal, forexample, a mobile phone, a smartphone, or a tablet terminal, and adigital camera have been widely used recently. Thus, there are moreopportunities to perform photographing of a digital image for people ingeneral.

When performing photographing by using a mobile terminal or an imagingdevice as described above, due to a depth of field which is determinedby characteristics of an optical system and a condition forphotographing, blur is caused in an object which is distant in a depthdirection from a main object which is in focus. Here, the depth of fieldrepresents an object-side distance range of an object which is in focusin an image. For example, in a case where a scene including both a nearview and a distant view is photographed under a condition that the depthof field is shallow, blur is caused in an object in the distant viewwhen an object in the near view is in focus, and blur is caused in theobject in the near view when the object in the distant view is in focus.

Accordingly, when a focal position is erroneously determined, an objectwhich is desired to be photographed by a user is not in focus and bluris caused. In a mobile terminal or an imaging device as described above,an auto focus (AF) function for automatically focusing on an object iswidely used, but an object is erroneously in focus by the AF in somecases. For example, there is a case where a focal position iserroneously determined in a background when photographing is performedby approaching an object at a short distance. There may also be a casewhere any objects which are photographed are not in focus and an imagein which all of them are blurring is provided.

Thus, it is desired to acquire a high-quality image having no blur andall objects in focus without focusing on only the object in the nearview or the object in the distant view.

PTL 1 described below proposes a technique of acquiring an image(hereinafter, also referred to as an “all-in-focus image”) having anenlarged depth of field by focusing on all objects from a near view to adistant view. With this technique, by moving a focusing lens in anoptical axis direction by a focusing lens driver, AF and focus-bracketphotographing are performed. The all-in-focus image is acquired based ona plurality of images acquired by consecutively photographing an objectwith focus bracketing photographing while changing a focal position.

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No. 2010-20758

SUMMARY OF INVENTION Technical Problem

A contrast system or a phase difference system has usually been used forthe AF. In any of the systems, an object at a certain point or objectsat some determined points in an image are in focus and an object withina depth of field including the object is in focus. Even when the depthof field is expanded in accordance with an optical system or a conditionfor photographing, there is a problem of deterioration in image qualitydue to a reduction in a light quantity. Accordingly, in a sceneincluding objects having greatly different distances in a depthdirection, with a conventional AF and optical system, it is difficult toacquire an all-in-focus image in which all objects are in focus.

With the technique disclosed in PTL 1, the all-in-focus image isacquired by using the AF and focus-bracket photographing, and a focalposition which is set at a time of the focus-bracket photographing isset such that every object in an image is in focus in at least any oneof images. It is noted that, however, when a certain part A in the imageis out of focus in all of a plurality of images, the part A remains outof focus in the all-in-focus image.

That is, it is considered that if the focal position of thefocus-bracket photographing is not set appropriately, even in a scenewhere objects exist at a plurality of distances, a part of objects isnot in focus and blur is caused, so that expansion of the depth of fieldbecomes insufficient.

Depending on an optical system to be used, a focal position may beshifted due to influence of the gravity when an orientation of animaging device is changed. However, such a point is not considered inthe technique described in PTL 1, and setting of the focal position maybecome inappropriate when the orientation of the imaging device ischanged. When there is such influence by the orientation, in order toacquire an all-in-focus image by the technique of PTL 1 regardless of anorientation state of the imaging device, it is necessary to performphotographing by setting many focal positions with fine steps so thatobjects at all distances are always in focus.

However, there is a problem that as the number of images to bephotographed increases accordingly, and a processing amount and a memorycapacity which are required for combining processing for theall-in-focus image increase. Further, since time required forphotographing becomes long with the increase in the number of images tobe photographed, influence by object shake, hand shake or the likeeasily appears and a difference between images which are photographed islikely to be great. Thereby, there is a possibility of deterioration inimage quality, such as a double image of an object.

The invention has been made in view of the aforementioned point, and anobject thereof is to easily acquire an image having an enlarged depth offield by appropriately setting a focal position and performingphotographing.

Solution to Problem

The invention has been made for solving the problem described above, andaccording to one aspect of the invention, provided is an imaging device,including: an image acquisition unit for acquiring a plurality of imageshaving different focal positions; an orientation acquisition unit foracquiring orientation information of the image acquisition unit; and animage processing unit for generating, from the plurality of images, animage having a depth of field which is enlarged compared to a depth offield of one of the plurality of images, in which a focal positionsetting value by which a focal position is determined is provided, andthe focal position setting value is corrected based on the orientationinformation.

Moreover, provided is the imaging device in which the orientationinformation has information about an angle formed by an optical axis ofthe image acquisition unit and a vertical direction, and a sign of acorrection amount with which the focal position setting value iscorrected is varied based on a sign of the angle information.

Moreover, provided is the imaging device in which the orientationinformation has information about an angle formed by an optical axis ofthe image acquisition unit and a vertical direction, and a correctionamount with which the focal position setting value is corrected isvaried based on magnitude of the angle information.

Moreover, provided is the imaging device in which the focal positionsetting value is determined based on a reference focal position settingvalue with which a reference object serving as a reference fordetermining the focal position is in focus, and a correction amount whenthe focal position setting value is corrected based on the orientationinformation varies in accordance with the reference focal positionsetting value.

Moreover, according to another aspect of the invention, provided is theimaging device in which based on the orientation information and thereference focal position setting value, a proportion of the number offocal position setting values with which an object closer to a near viewthan the reference object is in focus to the number of focal positionsetting values with which an object closer to a distant view than thereference object is in focus is adjusted, and based on the focalposition setting value which is adjusted, the image acquisition unitacquires the plurality of images.

The present specification includes the content in its entirety describedin the specification and/or the drawings of Japanese Patent ApplicationNo. 2013-164545 which is the base of the priority of the presentapplication.

Advantageous Effects of Invention

According to the invention, there is an advantage that an image havingan enlarged depth of field is able to be appropriately acquired byappropriately setting a focal position and performing photographing.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram illustrating an exemplaryconfiguration of an imaging device according to a first embodiment ofthe invention.

FIG. 2 illustrates an example of a photographed scene.

FIG. 3 illustrates an example of a relation between a lens drivingamount and a distance to an object.

FIG. 4 illustrates an example of a relation between inclination of anoptical axis and a distance to an object.

FIG. 5 is a flowchart illustrating a flow of processing of the presentembodiment.

FIG. 6 illustrates an example of a setting value of the driving amountin an example of a relation between a setting value of the lens drivingamount and a distance.

FIG. 7 illustrates an example of a relation between the setting value ofthe lens driving amount and the distance, and illustrates a variation incharacteristics when the optical axis has an angle.

FIG. 8 illustrates an example of a relation between the setting value ofthe lens driving amount and the distance, and illustrates a variation incharacteristics when the optical axis has an angle.

FIG. 9A is a functional block diagram illustrating an exemplaryconfiguration of an image processing unit.

FIG. 9B is a flowchart illustrating a flow of processing of an imageprocessing unit 13 of the invention.

FIG. 10 illustrates an example of a relation between the lens drivingamount and an in-focus distance when the imaging device is subjected toan impact.

DESCRIPTION OF EMBODIMENTS

Description will hereinafter be given in detail for embodiments of theinvention with reference to drawings. Expressions in the respectivefigures are exaggeratingly described to facilitate the understanding andare different from actual ones in some cases.

(First Embodiment)

FIG. 1 is a functional block diagram illustrating a schematicconfiguration example of an imaging device 1 according to a firstembodiment of the invention. The imaging device 1 includes an imageacquisition unit 10 for acquiring a plurality of pieces of imageinformation by performing photographing while changing a focal position,an orientation acquisition unit 11 for acquiring orientation informationof the image acquisition unit 10 or a whole of the imaging device 1including the image acquisition unit 10, an information storage unit 12in which, for example, setting information of a focal position isstored, and an image processing unit 13 for performing image processingfor image information output by the image acquisition unit 10 andcombining images having an enlarged depth of field. The imaging device 1in the present embodiment includes, for example, a processor such as aDSP (Digital Signal Processor) or a CPU (Central Processing Unit), and amain storage device such as a RAM (Random Access Memory), and is able torealize processing of each processing unit described above by executinga program stored in the storage device. Further, by including aprogrammable integrated circuit such as an FPGA (Field Programmable GateArray) or an integrated circuit dedicated for the aforementionedprocessing, the processing of each processing unit described above isalso able to be realized by hardware.

The image acquisition unit 10 includes an imaging element 100, such as aCCD (Charge Coupled Device) or a CMOS (Complementary Metal OxideSemiconductor), for converting received light to an electric signal toprovide image information, an optical system 101 such as a lens forconcentrating light from an object on the imaging element, a lensdriving unit 102 for driving the lens of the optical system 101 in anoptical axis direction to change a focal position, and a driving amountsetting unit 103 for setting a driving amount of the lens driving unit.The lens driving unit 102 drives the lens of the optical system 101based on a setting value of the driving amount, which is set by thedriving amount setting unit 103, and changes the focal position. Thatis, the setting value of the driving amount is a setting value of thefocal position, and the focal position is set when the setting value ofthe driving amount is set. Note that, the image acquisition unit 10includes an analog signal processing unit, an A/D (Analog/Digital)conversion unit, and the like, which are not illustrated, and outputsthe signal from the imaging element as the image information.

In addition, a zoom mechanism 14, an aperture mechanism 15, and an alarmunit 16 are included. Although these mechanisms are explicitlyillustrated in FIG. 1, the zoom mechanism 14 and the aperture mechanism15 are generally included, for example, in the image acquisition unit10.

The image acquisition unit 10 has an AF function of a system which isconventionally used, such as a contrast system or a phase differencesystem, and focuses on an object by the AF to acquire information of alens position at the time of focusing. In this case, with the AFfunction of the contrast system, the lens of the optical system 101 isdriven by the lens driving unit 102 to change a focal position, imageinformation of a plurality of focal positions is acquired to calculate acontrast of each of them, and a focal position at which a value of thecontrast is maximum is set as an in-focus position, thus making itpossible to focus on the object automatically.

In the present embodiment, an object serving as a reference(hereinafter, referred to as a “reference object”) is brought into focusto acquire in-focus information, and a plurality of focal positions areset with the in-focus information as a reference and a plurality ofimages are acquired.

The reference object may be specified by using the AF function describedabove or may be specified directly by a user. The in-focus informationis focal position information when an object is brought into focus, andis information of a lens driving amount when the reference object isbrought into focus. The in-focus information is output from the lensdriving unit 102 to the driving amount setting unit 103. When thereference object is specified with the AF function to acquire thein-focus information, for example, the reference object is able to bespecified by an AF spot. The AF spot is set at a two-dimensionalcoordinate indicating a range of a field angle for photographing. Forexample, when a vicinity of the center of the field angle is set as theAF spot, an object at a position of the spot (for example, vicinity ofthe center of an image) is in focus. Further, when AF spots are set at aplurality of positions, an object at a focal position closest to a nearview side among objects at the spot positions is able to be in focus.Alternatively, it is possible to acquire in-focus information byspecifying an object at a focal position of being most frequentlybrought into focus from in-focus information of the plurality of AFspots.

The user is able to acquire in-focus information by specifying an objectdirectly by using a display device such as a liquid crystal display oran organic EL (Electro Luminescence) display or a touch panel of acapacitance type or an electromagnetic induction type for the imagingdevice. For example, in a case of the imaging device including thedisplay device, the user is able to specify an object to set as areference object from a preview image, which is displayed on the displaydevice, with a button or the like included in the imaging device. Inaddition, in a case of the imaging device including the display deviceand the touch panel, by directly touching an object on a display inwhich a photographed image is displayed for previewing, the user is ableto specify the object to be in-focus.

Any position may be set in advance in a view angle of the imaging deviceto set an object which exists at the position as the reference object.

When a distance from the imaging device to an object is known, an objectwhich is at any distance set in advance may be set as the referenceobject.

Further, it may be configured so that any object such as a person, ananimal or a building is registered in advance, and an object is detectedwith a use of a known image recognition technique, for example, a facialrecognition technique or an object recognition technique and set as thereference object.

Based on the in-focus information of the lens driving unit 102 and theorientation information of the orientation acquisition unit 11, thedriving amount setting unit 103 reads setting information of the settingvalue of the driving amount from the information storage unit 12 to setthe setting value of the lens driving amount. Note that, the drivingamount setting unit 103 is provided for easy understanding of thedescription, and the setting value of the driving amount may be setinside the lens driving unit 102 or may be set by providing a settingunit outside the image acquisition unit 10 separately.

The lens of the optical system 101 includes a focusing lens for changinga focal position, or the like, and may be configured to have one or morelenses.

The lens driving unit 102 drives the lens in order to change the focalposition, and is composed of an actuator of a VCM (Voice Coile Motor)system or an SMA (Shape Memory Alloy) system used in general smartphonesand table terminals. Here, the VCM is a system in which the lens isoperated by electromagnetic force using a magnet and a coil, and the SMAis a system in which the lens is made up and down by energizingshape-memory alloy for heating. Since influence of the gravity isapplied when the lens is driven with such driving systems, the lensdriving amount changes in accordance with the influence of theorientation of the imaging device and the focal position changes in somecases. An advantage of the imaging device according to the presentembodiment is achieved in a terminal and a device which employ, inparticular, the systems as described above. Note that, when the focalposition changes in accordance with the orientation of the imagingdevice, that is, in accordance with the influence of the gravity, forexample, a liquid crystal lens and the like may be used withoutlimitation to the optical system and the driving systems as describedabove. In this case, as the liquid crystal lens, there are one forchanging the focal position with an electric control signal, one formechanically changing the focal position by an actuator, and the like.

The orientation acquisition unit 11 is composed of a tri-axialaccelerating sensor mounted in general mobile equipment such as asmartphone, and measures orientation information of the imageacquisition unit 10 or the whole of the imaging device 1 including theimage acquisition unit 10. The orientation information is angleinformation based on an optical axis of the optical system 101 includedin the image acquisition unit 10 and an axis in a vertical directionwhich is a direction in which the gravity acts. The angle informationwhen the optical axis and the axis in the vertical direction are matchedis −90 degrees and the angle information when the orientation of theimaging device is horizontal and the optical axis is parallel to theground, that is, when the optical axis and the axis in the verticaldirection form a right angle is 0 degree. In this manner, a sign of theangle information is changed with a time when the optical axis ishorizontal to the ground being a boundary. That is, when the angleinformation is negative, the optical axis has a depression angle and theimaging device has the orientation of being held by inclining downward.To the contrary, when the angle information is positive, the opticalaxis has an elevation angle and the imaging device has the orientationof being held by inclining upward.

Here, as the tri-axial accelerating sensor, a sensor of a capacitancetype, a piezo-resistive type, a thermal sensing type or the like inwhich MEMS (Micro Electro Mechanical Systems: a micro electro mechanicalelement and a creating technology thereof) is applied is used. Thecapacitance type is a system in which a change in a capacity between amovable part and a fixed part of a sensor element is detected, and thepiezo-resistive type is a system in which deformation of a springportion caused by acceleration is detected with a use of apiezo-resistance element equipped in the spring portion connecting amovable part and a fixed part of a sensor element. Further, the thermalsensing type is a system in which thermal air current is generated in ahousing by a heater and a change in convection caused by acceleration isdetected as a thermal resistance or the like.

A method for measuring an orientation is not limited to the method bythe tri-axial accelerating sensor, and any method may be used as long asorientation information is able to be estimated. The MEMS is suitablyused because the device is able to be miniaturized.

Note that, the orientation information may be acquired by estimating theorientation of the imaging device from object information of an image.The orientation may be estimated, for example, by using information ofthe ground, or information of an object, such as a building, which isvertical to the ground. In this case, the image processing unit may beprovided in such a manner that the orientation is estimated at theorientation acquisition unit 11 based on image information of the imageacquisition unit 10, or the orientation may be estimated by the imageprocessing unit 13 without providing the orientation acquisition unit 11to output orientation information to the driving unit setting unit 103.

The information storage unit 12 is composed of, for example, a flashmemory, a hard disc or the like, and stores a focal position at a timeof photographing, that is, setting information of the setting value ofthe lens driving amount, and the like therein. It is desired that theinformation is stored in the information storage unit 12 in advance, butit may be configured so that the information is directly input by theuser before photographing.

The image processing unit 13 is composed of a processor such as, forexample, a DSP or a CPU, and performs image processing for a pluralityof pieces of image information input from the image acquisition unit 10to thereby combine images having an enlarged depth of field. Theprocessing will be described below in detail.

Next, schematic description for a method for acquiring images having anenlarged depth of field from a plurality of images having differentfocal positions will be given below with reference to drawings.

FIG. 2 exemplifies a scene where two objects of an object in a near view2 a and an object in a distant view 2 b exist. In the scene of FIG. 2, adistance between the two objects 2 a and 2 b is great in a depthdirection, and in a case where photographing is performed by the imagingdevice with a narrow depth of field, when one object is in focus, bluris caused in the other object. Note that, when simply described as a“distance” in the following description, it represents a distance in thedepth direction when viewed from the imaging device.

Then, photographing is performed while changing a focal position, and animage in which the object in the near view 2 a is in focus and an objectin which the object in the distant view 2 b is in focus are acquired.Next, by selecting and combining focused pixels of the respectiveimages, an image in which both objects in the near view and the distanceview are in focus is able to be acquired.

However, it is not clear at which distance each object exists in anactual photographed scene, and there is also a possibility that two ormore objects exist at various distances. Accordingly, it is necessary toacquire a plurality of images while changing the focal position so thatall the objects are in focus. In this case, by determining the focalposition at a time of photographing in consideration of, for example,characteristics of the optical system, all the objects are able to be infocus with the minimum number of photographs. It is desirable that itbecomes possible to perform photographing with an optimum focal positionand number of photographing and suppress a processing amount, a memorycapacity, or power consumption.

FIG. 3(a) illustrates an example of a relation between the lens drivingamount when the optical system 101 is driven by the lens driving unit102 and the focal position is changed, and a distance at which focus isachieved. FIG. 3(a) indicates the relation for a case where a distantview is in focus when the lens driving amount is small and a near viewis in focus when the lens driving amount is large. The lens drivingamount as a horizontal axis in FIG. 3 indicates the lens driving amountfrom a lens position at which an object at infinity is able to be infocus, by setting the position as a reference. The distance as avertical axis indicates a distance from the imaging device to an object.Each curve in FIG. 3 illustrates the distance at which focus is achievedwith a certain lens driving amount, and a portion between two curves Aand B illustrated in FIG. 3 indicates a distance range (hereinafter,referred to as an “in-focus range”) in which focus is allowed withoutcausing blur. FIG. 3(a) indicates that the in-focus range is ar in acase of a lens driving amount a1. In this case, focus and non-focus areable to be set as appropriate, for example, by values of a pixel size ofan imaging element and a point-spread function.

In a case of characteristics of FIG. 3(a), when three images arephotographed by setting three lens driving amounts b1 to b3 as indicatedin FIG. 3(b), the focused range ar shifts to br and a depth of field isable to be enlarged. Since the characteristics indicated in FIG. 3(a)vary in accordance with device characteristics of the lens driving unit102, characteristics of the optical system, a condition forphotographing, and the like, by determining the setting value of thelens driving amount in consideration of them, it is possible to combinean all-in-focus image having an enlarged depth of field with the optimumnumber of photographs. Note that, the lens driving amount is set by thedriving amount setting value, and the driving amount setting value is anumerical value indicating a current amount required for driving thelens. Note that, the characteristics indicated in FIG. 3 are an example,and characteristics that a distance at which focus is achieved is on adistant view side as the lens driving amount increases, orcharacteristics which vary linearly are also considered.

Since the optical system 101 and the lens driving unit 102 areinfluenced by the gravity, it is necessary to consider that the focalposition changes depending on the orientation of the imaging device 1.

For example, characteristics of the distance at which focus is achievedwhen the setting value of the lens driving amount is fixed and theorientation of the imaging device 1 is changed are as illustrated inFIG. 4.

A horizontal axis of FIG. 4 indicates angle information representinginclination of the optical axis and a vertical axis of FIG. 4 indicatesa distance from the imaging device 1 to an object. A curve of FIG. 4illustrates the distance at which focus is achieved at a certain angle,and illustrates characteristics of a center value of the in-focus range.The characteristics illustrated in FIG. 4 indicate that the distance atwhich focus is achieved changes to a near view side when the angleinformation changes in a minus direction and the degree of the elevationangle becomes great. Similarly, it is indicated that the distance atwhich focus is achieved changes to a distant view side when the angleinformation changes in a plus direction and the degree of the depressionangle becomes great. Note that, the angle information indicating theinclination of the optical axis of the optical system 101 is angleinformation of the image acquisition unit 10 including the opticalsystem 101, and inclination of the image acquisition unit 10 is angleinformation of the imaging device 1 including the image acquisition unit10. Note that, the characteristics illustrated in FIG. 4 are an example,and the distance at which focus is achieved may change in a reversedmanner between the plus direction and the minus direction of the changein the angle information. That is, the distance at which focus isachieved may be on the near view side when the degree of the elevationangle becomes great, and the distance at which focus is achieved may beon the distant view side when the degree of the depression angle becomesgreat. Moreover, the characteristics may change linearly. As describedabove, even when the setting values of driving amount are equal, thedistance at which focus is achieved changes in accordance with theorientation of the imaging device. A method for setting the settingvalue of the lens driving amount will be described below in detail.

As described above, the imaging device 1 in the present embodiment setsthe focal position at a time of photographing by consideringcharacteristics of the optical system, a condition for photographing,and the orientation of the imaging device.

Next, a flow of processing of the imaging device 1 in the presentembodiment will be described below with reference to FIG. 5. FIG. 5 is aflowchart indicating a flow of the processing of the imaging device 1.

First, when the processing starts, the image acquisition unit 10 focuseson an object with the method described above, and acquires a lensdriving amount when focus is achieved as in-focus information (stepS101).

The imaging device 1 then acquires orientation information of theimaging device 1 by the orientation acquisition unit 11 (step S102). Theorientation information indicates information about inclination of theimage acquisition unit 10 or the whole of the imaging device 1 includingthe image acquisition unit 10 to the ground, that is, inclinationinformation of the optical axis of the optical system 101.

When photographing is performed actually, step S101 and step S102 areable to be performed as follows. For example, when a terminal includinga display device and a touch panel, such as a smartphone or a tabletterminal, is used, it is desirable to perform photographing by acquiringin-focus information when a user touches an object indicated in an imagedisplayed for previewing on the display device, and measuring anorientation when releasing a shutter. Thereby, the object desired by theuser is in focus and photographing is performed with a setting accordingto the orientation when the shutter is released, thus making it possibleto correspond to even a case where the orientation of the terminalchanges after the touch panel is touched. Moreover, in a case of animaging device including a shutter button, such as a digital camera, anobject is in focus by a half-pressing operation of the shutter buttonand the shutter is released with a fully pressing operation. In thiscase, the orientation may change before the shutter button shifts to afully-pressed state from a half-pressed state, so that it is desirableto estimate the orientation when the shutter button is fully pressed.

Next, the driving amount setting unit 103 reads information from theinformation storage unit 12 based on the in-focus information and theorientation information which are described above, and sets a settingvalue of the driving amount (step S103). Step S103 will be describedbelow in detail.

The image acquisition unit 10 then drives the lens based on the settingvalue of the driving amount by the driving amount setting unit 103 toperform photographing, and acquires a plurality of pieces of imageinformation (step S104).

Finally, the image processing unit 13 performs image processing for theplurality of pieces of image information from the image acquisition unit10, and combines images having an enlarged depth of field and outputs aresultant image (step S105). Step S105 will be described in detailbelow.

The image output at step S105 may be displayed on a display device suchas a liquid crystal display or an organic EL display, or may be storedin a storage device such as a flash memory or a hard disc. In addition,the plurality of pieces of image information acquired at the imageacquisition unit 10 may be also stored in the storage device. Theimaging device 1 may be configured to include these devices.

(Step S103: Method for Setting a Setting Value of the Lens DrivingAmount)

The method for setting the setting value of the lens driving amount inthe driving amount setting unit 103 (step S103) will be described below.

According to the in-focus information of step S101 and the orientationinformation of step S102, the driving amount setting unit 103 readsinformation from the information storage unit 12 and performs setting.

For example, information of the setting value of the driving amountcorresponding to any orientation state of the imaging device 1 and aposition of any reference object is stored in the information storageunit 12 in advance, and the optimum setting value of the driving amountis read and set in accordance with the orientation information and thein-focus information. When the orientation information and the in-focusinformation change, for example, when the orientation changes in themiddle of photographing or the reference object moves, the informationis red from the information storage unit 12 as appropriate. Theinformation of the setting value of the driving amount, which is storedin the information storage unit 12, is obtained with an optimum settingvalue of the driving amount, for example, when the imaging device 1 hasa certain fixed orientation and a reference object exists at a certaindistance as a reference. The setting value which is set as the referenceis able to be obtained in advance by correction in consideration ofcharacteristics by the orientation and characteristics when a positionof the reference object changes as indicated in FIG. 4.

One example of the method for setting the setting value of the drivingamount will be described below with reference to an example illustratedin FIG. 6. FIG. 6(a) illustrates a relation between the setting value ofthe lens driving amount and the distance when the imaging device havingcertain characteristics of a depth of field has a certain orientation. Ahorizontal axis of FIG. 6 indicates the setting value of the lensdriving amount and a vertical axis indicates the distance from theimaging device to the object. Each curve illustrated in FIG. 6illustrates the distance at which focus is achieved by the lens which isdriven with a certain setting value of the driving amount, and anin-focus range is provided between the curves C and D.

First, the driving amount setting unit 103 acquires a setting value ofthe driving amount with which a reference object is in-focus. Thesetting value of the driving amount with which the reference object isin-focus is able to be obtained from the in-focus information acquiredat step S101. Specifically, since the lens driving amount when thereference object is in-focus is found from the information of the lensdriving unit 102, with the setting value of the driving amount at thattime, the setting value of the driving amount for the reference objectis able to be known. FIG. 6(a) indicates that the setting value of thedriving amount with which the reference object is in-focus is c0. Basedon the setting value of the driving amount c0 and the orientationinformation at step S102, the setting values of the driving amount areread from the information storage unit 12. That is, the setting valuesof the driving amount corresponding to the position of the referenceobject and the orientation of the imaging device 1 are read. In thepresent example, setting values of the driving amount c1 and c2 are redfrom the information storage unit 12 as illustrated in FIG. 6 (b). Here,the setting value of the driving amount with which the reference objectis in focus, such as c0 above, is called a reference setting value ofthe driving amount. Note that, since the setting value of the drivingamount is a setting value of a focal position as described above, thereference setting value of the driving amount is a reference settingposition of the focal position.

Here, as indicated in FIG. 6(b), focus is achieved at c1 and c2 in adistance range before and after the in-focus range cr0 of c0 and thein-focus ranges cr0, cr1 and cr2 are set so as to be overlapped witheach other and stored in the information storage unit 12. Thereby, thein-focus range is not interrupted and it is possible to prevent that anobject which is in focus and an object which is not in focuscontinuously appear in turn. For example, it is possible to prevent thatwhen an object which continues in the depth direction, such as theground, is photographed, an in-focus range and a non-focus range appearin the middle of the object and image quality is deteriorated.Accordingly, it is desirable to store the setting value of the drivingamount with which the in-focus range continues in the informationstorage unit 12 in advance.

When objects exist at only two distances in the near view and thedistant view as the example of FIG. 2, however, it is only required thatfocus is achieved at the two distances and it is not always requiredthat the in-focus range continues. That is, if there is no influencesuch as deterioration of image quality, the setting value of the drivingamount with which the in-focus range becomes discontinuous may be set.

In the above case, when the reference object exists in an intermediateview and the reference setting value of the driving amount is c0, c1 atwhich focus is achieved in the near view and c2 at which focus isachieved in the distance view are set as other setting values of thedriving amount. For example, when the reference object exists in thenear view, the reference setting value of the driving amount is c1, andc0 at which focus is achieved in the intermediate view and c2 at whichfocus is achieved in the distant view are set. Similarly, when thereference setting value of the driving amount is c2, c1 at which focusis achieved in the near view and c0 at which focus is achieved in theintermediate view are set. For example, even if the setting value of thedriving amount with which focus is achieved on a further distant side isset when the reference setting value of the driving amount is a valuewith which focus is achieved in the distant view, an effect of enlargingthe depth of field is not sufficiently achieved. Thus, it is desirableto decide the distance to the reference object with the in-focusinformation as the near view, the intermediate view, the distant view,or the like in this manner and determine to which side of the near viewside and the distant view side from the reference object the focalposition is to be changed for photographing, because the depth of fieldis able to be enlarged appropriately, which is thus desirable. Moreover,it is assumed that the number of photographed images is three or moredepending on characteristics of the depth of field or the orientationstate of the imaging device. In this case, it is set that the depth offield is able to be enlarged appropriately by changing a proportion ofthe number of photographed images of the near view side to the number ofphotographed images of the distant view side from the reference object.For example, when the reference object exists in the intermediate view,the proportion is changed, for example, by increasing the number for thenear view side and decreasing the number for the distant view side byconsidering that the near view side has the narrower depth of field. Bystoring the information thereof in the storage information unit 12 inadvance, it is possible to perform setting by reading based on thein-focus information and the orientation information. This is desirablebecause it is thereby possible to perform photographing with theappropriate focal position and the appropriate number of photographedimages according to the orientation of the imaging device and theposition of the reference object and to enlarge the depth of field.

For judging the distance as the near view, the intermediate view, thedistant view or the like, for example, by setting a distance thresholdin advance, the distance is able to be decided based on the in-focusinformation and the distance threshold. With information about the lensdriving amount which is the in-focus information and information about aview angle, a focal distance, a pixel size and the like of the opticalsystem, a rough distance value from the imaging device to the referenceobject is able to be calculated. When the distance value to thereference object is smaller than the distance threshold used fordecision as a near view, which is set in advance, it is decided that thereference object exists in the near view. At this time, by setting atleast one setting value of the driving amount to be a value with whichfocus is achieved at a distance longer than the distance threshold inaddition to the reference setting value of the driving amount, it ispossible to focus on the reference object and an object closer to thedistant view than the reference object. Similarly, when the distancevalue to the reference object is larger than the distance threshold usedfor decision as a distant view, which is set in advance, it is decidedthat the reference object exists in the distant view. At this time, bysetting at least one setting value of the driving amount to be a valuewith which focus is achieved at a distance shorter than the distancethreshold in addition to the reference setting value of the drivingamount, it is possible to focus on the reference object and an objectcloser to the near view than the reference object. The number and valuesof the distance thresholds may be set to any number and values and arenot limited to the aforementioned two distance thresholds. When the twodistance thresholds used for decision as the near view and decision asthe distance view are set, three distance ranges of the near view, theintermediate view and the distant view are able to be set. In theexample of FIG. 6, the original depth of field which is determined inaccordance with the characteristics of the optical system and thecondition for photographing is expanded and an interval between thecurves C and D is wide. Therefore, it is possible to acquire an imagehaving an enlarged depth of field by photographing three images, and bysetting the two distance thresholds and the three distance ranges of thenear view, the intermediate view and the distant view, the setting valueof the driving amount is able to be set so that focus is achieved ateach of the distance ranges. Moreover, when the depth of field isnarrower than that of the example of FIG. 6, for example, it is requiredto increase the number of the setting values of the driving amount to beset and increase the number of photographed images. Thus, it isdesirable that the distance threshold is set in accordance with thenumber of photographed images. In this manner, by determining thedistance threshold and the setting value of the driving amount accordingto the characteristics determined by the field of depth of the imagingdevice, the depth of field is able to be enlarged appropriately with theminimum number of photographs. As described above, by storing thesetting value of the driving amount corresponding to the any orientationstate of the imaging device in the information storage unit 12 andreading and setting the setting value of the driving amount based on thein-focus information and the orientation information, the setting valueof the driving amount is able to be set appropriately.

Note that, though the information stored in the information storage unit12 is read to perform setting in the above, it may be configured so thatonly information serving as a reference and correction information arestored without storing all the information in advance so as to performcorrection based on the in-focus information and the orientationinformation.

For example, when the orientation of the imaging device is horizontaland the angle information is 0 degree, the reference setting value ofthe driving amount with which a reference object at a certain distanceis in focus and other setting values of the driving amount determined inaccordance with the reference setting value of the driving amount arestored in the information storage unit 12 in advance as the referenceinformation. Further, information about a correction amount and acorrection direction when a position of the reference object changes andwhen the orientation changes are stored in the information storage unit12.

The driving amount setting unit 103 reads the information about thecorrection amount and the correction direction from the informationstorage unit 12 based on the in-focus information of step S101 and theorientation information of step S102, and based on the information,corrects the reference information stored in advance. Since thereference information includes the reference setting value of thedriving amount and other setting values of the driving amount, each ofthe setting values of the driving amount is corrected.

Here, for example, when the imaging device has the inclined orientationand the angle information changes, the relation between the settingvalue of the lens driving amount and the distance changes as indicatedin FIG. 7. A horizontal axis of FIG. 7 indicates the setting value ofthe lens driving amount and a vertical axis indicates the distance fromthe imaging device to an object. Each curve of FIG. 7 illustrates thedistance at which the lens is driven with a certain setting value of thedriving amount and focus is achieved at a focal position at that time,and illustrates characteristics of a center value of the in-focus range.FIG. 7(a) illustrates characteristics when the optical axis ishorizontal to the ground and the angle information is 0 degree. In acase where the relation between the inclination of the optical axis andthe distance is like the characteristics indicated in FIG. 4, therelation between the setting value of the lens driving amount and thedistance when the optical axis is inclined changes as FIG. 7(b) and FIG.7(c). FIG. 7(b) illustrates characteristics in a case of a state wherethe angle information changes in the plus direction and the optical axishas an elevation angle, and FIG. 7(c) illustrates characteristics in acase of a state where the angle information changes in the minusdirection and the optical axis has a depression angle. The curveindicating the characteristics of FIG. 7(b) indicates that the curve ofFIG. 7(a) shifts to the distant view side. The curve indicating thecharacteristics of FIG. 7(c) indicates that the curve of FIG. 7(a)shifts to the near view side. Note that, the characteristics indicatedin FIG. 7 are an example and the shifting direction of thecharacteristics as indicated in FIG. 7(b) and FIG. 7(c) may be reversed.That is, it is considered that the shifting direction is reversedbetween the case where the optical axis has the elevation angle and thecase where it has the depression angle. A shifting amount changes inaccordance with a degree of the inclination of the orientation, and theshifting amount also increases as the imaging device is inclined greatlyand the angle information becomes great.

When the characteristics change depending on the orientation as FIG. 7,the setting value of the driving amount is corrected as indicated inFIG. 8. Similarly to FIG. 7, a horizontal axis indicates the settingvalue of the driving amount and a vertical axis indicates the distancefrom the imaging device to an object in FIG. 8. Each curve illustratedin FIG. 8 represents the distance at which focus is achieved at a focalposition of the lens which is driven with a certain setting value of thedriving amount, and an in-focus range is indicated by two curves.Similarly to FIGS. 7(a), (b), and (c), FIGS. 8(a), (b), and (c)respectively indicate characteristics when the orientation ishorizontal, characteristics when the optical axis has the elevationangle, and characteristics when the optical axis has the depressionangle. In FIGS. 8(b) and (c), the solid curves represent respectivecharacteristics and dotted curves represent the characteristics of FIG.8(a). Arrows in graphs of FIGS. 8(b) and (c) indicate directions of thechange in the characteristics. In a case of an example of FIG. 8,setting values of the driving amount d0, d1 and d2 indicated in FIG.8(a) are stored as reference information in the information storage unit12. At this time, when the reference object exists in the intermediateview, d0 is the reference setting value of the driving amount. Further,information about a correction amount and a correction directionaccording to orientation information is stored in the informationstorage unit 12, and based on the information, d0, d1 and d2 arecorrected, and setting values of the driving amount e0, e1 and e2 ofFIG. 8(b), and setting values of the driving amount f0, f1 and f2 ofFIG. 8(c) are provided.

FIG. 8 illustrates an example in which as the characteristicsillustrated in FIG. 4, the focal position shifts to the distant viewside when the angle information changes in the plus direction, and thefocal position shifts to the near view side when the angle informationchanges in the minus direction. When the characteristics shift to thedistant view side as illustrated in FIG. 8(b), the setting value of thedriving amount is corrected to the plus direction in which the valueincreases, and when the characteristics shift to the near view side asillustrated in FIG. 8(c), the setting value of the driving amount iscorrected to the minus direction in which the value decreases. In thismanner, the direction in which the setting value of the driving amountis corrected (sign of the correction amount) is varied in accordancewith the direction (sign) in which the angle information changes. Forexample, in contrast to the characteristics illustrated in FIG. 4, thefocal position shifts to the near view side upon changing of the angleinformation in the plus direction and the focal position shifts to thedistant view side upon changing of the angle information in the minusdirection, the direction in which the setting value of the drivingamount is corrected is also reversed. This makes it possible to set thesetting value of the driving amount correspondingly to the inclinationdirection of the orientation and the changing direction of thecharacteristics appropriately.

In a general lens driving system as assumed in the present embodiment,the distance at which focus is achieved changes greatly as theinclination of the optical axis increases as illustrated in FIG. 4.Accordingly, as the angle information changes greatly, the shiftingamount of the characteristics illustrated in FIG. 7 and FIG. 8 alsoincreases. When the shifting amount increases in this manner, thecorrection amount for the setting value of the driving amount alsoincreases. When the correction amount for the setting value of thedriving amount is also varied in accordance with the magnitude of thechange of the angle information, it becomes possible to appropriatelyset the setting value of the driving amount.

As illustrated in FIG. 8, d0, d1 and d2 serving as the referenceinformation are corrected with the respectively different correctionamounts. This is because the in-focus range varies depending on thesetting value of the driving amount and the characteristics thereof varydepending on the imaging device. For example, when all of d0, d1 and d2are corrected with the fixed correction amount, for example, thein-focus range of each setting value of the driving amount is deviatedto the near view side or the distant view side, and it becomes difficultto appropriately enlarge the depth of field. Moreover, the setting valueof the driving amount serving as the reference information changes inaccordance with the position of the reference object. That is, inaccordance with the reference setting value of the driving amount whenthe reference object is in focus, other setting values of the drivingamounts also change. Thus, it is desirable to vary the correction amountfor each setting value of the driving amount in accordance with thevalue of the reference setting value of the driving amount and thecharacteristics of the depth of field that the imaging device has,because it is possible to set the appropriate setting value of thedriving amount.

As described above, by correcting the reference information of thesetting value of the driving amount with the correction amount and thecorrection direction in accordance with the in-focus information and theorientation information, it is possible to set the setting value of thedriving amount appropriately.

(Step S105: Image Processing Method)

Processing procedure and a processing method of the image processingunit 13 will be described below with reference to a drawing. FIG. 9A isa functional block diagram illustrating an exemplary configuration ofthe image processing unit 13. FIG. 9B is a flowchart indicating a flowof processing of the image processing unit 13.

As illustrated in FIG. 9A, the image processing unit 13 has an imagegroup positioning processing unit 13-1, an each-image in-focus degreecalculation unit 13-2, an in-focus degree comparison unit 13-3, and animage composition unit 13-4.

First, the image group positioning processing unit 13-1 of the imageprocessing unit 13 performs positioning for a plurality of pieces ofimage information input from the image acquisition unit 10 (S1051). Whenphotographing is performed by changing the focal position likefocus-bracket photographing, a view angle changes for each image and aposition of an object is shifted, so that positioning needs to beperformed between images.

The positioning for the images is able to be performed by obtaining anamount of the change in the view angle based on the lens driving amount,that is, the focal position by using the setting value of the lensdriving amount set at preceding step S103 and performing processing forenlarging and reducing the images. Note that, the processing forenlarging and reducing the images is able to be performed by using ageneral interpolation method such as bilinear interpolation or bicubicinterpolation.

When it is difficult to perform the positioning only by simpleenlargement and reduction processing, the positioning may be performedby calculating an amount of movement of a feature point between theimages, and performing image processing by affine transformation such asenlargement and reduction, rotation and translation based on the amountof movement. The feature point is able to be detected by using a generalmethod for detecting a feature point, such as a Harris corner detectionmethod, and the same feature point is thereby detected from each image.Moreover, by calculating similarity between a target pixel of one imageand a pixel on the other image corresponding thereto with a calculationmethod such as SAD (Sum of Absolute Difference) or SSD (Sum of SquaredDifference), the amount of movement of the feature point is able to becalculated.

Note that, the positioning is performed by using any one image among aplurality of images which are photographed as a reference and matchingan object position on the reference image and object positions on otherimages. The reference image is able to be set freely. However, since animage having a wide view angle includes a range which is notphotographed in an image having a narrow view angle, it is desirablethat an image on the near view side, which has the smallest view angle,is used as the reference. Since enlargement processing in whichinterpolation of a pixel having no information is performed causesgreater deterioration of image quality compared to reduction processing,it is desirable to perform positioning by reduction processing for animage having a wide view angle with an image having a narrow view angleas a reference as described above because it is possible to suppressdeterioration of image quality.

Next, the each-image in-focus degree calculation unit 13-2 of the imageprocessing unit 13 calculates an in-focus degree for each pixel withrespect to each image subjected to positioning (S1052). The in-focusdegree represents a degree at which an object is in focus, and is ableto be obtained by utilizing that a region which is in focus has a highercontrast. For example, a difference between a maximum value and aminimum value of a pixel value in a rectangular region with a targetpixel as a center is calculated as a contrast value and thus able to beused as the in-focus degree of the target pixel. Note that, the pixelvalue is a luminance value of the pixel, an RGB value or the like. Notethat, the in-focus degree is not limited thereto, and sharpness or thelike may be calculated by using a degree of expansion of an edge part ofan object and may be obtained in any method as long as in-focus degreesare able to be compared between images.

Next, the in-focus degree comparison unit 13-3 of the image processingunit 13 compares the in-focus degrees calculated for each image at allpixel positions of the images after positioning (S1053), and the imagecomposition unit 13-4 of the image processing unit 13 combines theimages by weighted average so that weight of a pixel of the image havingthe highest in-focus degree is large (S1054). Thereby, an all-in-focusimage obtained by combining only the pixels having the highest in-focusdegree is acquired.

Note that, in the composition of the all-in-focus image, a pixel of animage having the highest in-focus degree may be simply selected for eachpixel position or a weighted average value based on the in-focus degreesmay be calculated. For example, in a case of being in a region of anobject which is characteristic, the in-focus degree is likely to be thehighest in any one of photographed images, and weight of a pixel of theimage having the high in-focus degree may be merely increased to be setas a pixel of the all-in-focus image. However, for example, in a flatregion of an object which is less characteristic, the in-focus degreedoes not change in all the images and similar values are possiblyprovided. In this case, if the pixel of the image having the highestin-focus degree is selected based on a slight difference between thein-focus degrees, for example, even in a case of a flat region of thesame object, a pixel of a different image is selected at an adjacentpixel position and deterioration of image quality may be caused in thecombined image. Thus, the pixel value of the all-in-focus image may becalculated by performing weighted average for the pixels values of therespective images with a coefficient having a small inclination ofweight. Further, it may be configured so that a coefficient of weight iscalculated by comprehensively judging in-focus degrees of the targetpixel and neighboring pixels thereof and a pixel value at a position ofthe target pixel is calculated by weighted average.

With the method described above, the imaging device 1 of the presentembodiment sets the setting value of the lens driving amount accordingto the orientation information and the in-focus information. Byacquiring a plurality of images while changing the focal position basedon the setting value of the driving amount, which is set, and performingimage processing based on the images, it is possible to combine theimages having an enlarged depth of field.

That is, also when the orientation of the imaging device 1 changes, theimaging device 1 of the present embodiment is able to appropriately setthe focal position and appropriately acquire an image having an enlargeddepth of field. In consideration of characteristics of the depth offield determined based on the optical system and a condition forphotographing, and based on the in-focus information to the referenceobject, the focal position is set, so that the reference object isalways in focus and an image having an enlarged depth of field is ableto be acquired by suppressing the number of images to be photographed toa minimum. This makes it possible to suppress a processing amount, amemory capacity and power consumption which increase in accordance withthe number of images.

(Second Embodiment)

In the imaging device 1 described in the first embodiment above, it isassumed that a single-focus lens is used for the lens of the opticalsystem 101 and a focusing lens is driven by the lens driving unit 102 tothereby change a focal position.

Assumed in the second embodiment of the invention is a case where anoptical system capable of zooming on an object is included. The imagingdevice of the present embodiment includes a zoom mechanism 14 in theconfiguration of the imaging device 1 of FIG. 1.

For example, when a varifocal lens which changes the focal positionsimultaneously with zooming is included as the zoom mechanism 14,characteristics of the change in the focal position according to adriving amount of the varifocal lens are acquired in advance. Similarlyto the embodiment described above, the setting value of the drivingamount according to the characteristic information is stored ininformation storage unit 12 and orientation information and theinformation according to the driving amount of the varifocal lens areread, so that photographing at the optimum focal position is able to beperformed.

When a zoom lens capable of performing zooming without changing thefocal position is included, characteristics according to a drivingamount of the zoom lens are acquired in advance. As has been describedin the embodiment above, the lens driving amount and the characteristicsof the distance at which focus is achieved change depending on theorientation. Further, since the characteristics change as the zoom lensoperates, by storing information of the setting value of the drivingamount according to the characteristics in the information storage unit12, photographing is able to be performed at the optimum focal positioncorresponding to the characteristics of the zoom lens.

Note that, both of the lenses above have a view angle narrowed whenzooming, so that an object to be photographed may be limited. Forexample, while a scene including objects in a near view and in a distantview is being photographed with a wide angle, when the object in thedistant view is zoomed, there is a case where the object in the nearview side is out of a frame and there is only the object in the distantview within a view angle. That is, in a case of a state where a zoomedamount is large, when a distance of an object, which is found from thein-focus information of step S101, is on the distant view side, theobject in the distant view side is zoomed and there is more likely to beno object in the near view side. In this case, since the object in thenear view side is not photographed, it is not necessary to performphotographing by allowing a focal position to be located in the nearview. Similarly, when the object in the near view is zoomed, the distantview is less likely to be included. Thus, it may be configured such thatthe setting value of the lens driving amount according topresence/absence of zooming or a degree of zooming in the informationstorage unit 12, and further, the setting value of the driving amount isset in accordance with information of the distance of the object, whichis obtained from the in-focus information, i.e., information of the nearview, the intermediate view, the distant view and the like. This makesit possible to acquire an image having an enlarged depth of field withthe optimum focal position and number of photographed images accordingto the information about zooming and the in-focus information.

Note that, though description has been given above by assuming a casewhere the setting value of the driving amount according to theinformation about zooming is stored in the information storage unit 12,information of a correction amount according to the information aboutzooming may be stored in the information storage unit 12. It may beconfigured so that the correction amount is thereby read based on theinformation about zooming and the setting value of the driving amount,which is set as a reference, is corrected.

With the method described above, since the imaging device of the presentembodiment sets the setting value of the lens driving amount inconsideration of the characteristics of the optical system including thezoom mechanism, it becomes possible to appropriately set the focalposition and perform photographing to appropriately acquire an imagehaving an enlarged depth of field even when the characteristics changedue to zooming.

Further, by setting the setting value of the lens driving amount inconsideration of information about a degree of zooming orpresence/absence of zooming or the like and the in-focus information, itis possible to acquire an image having an enlarged depth of field withthe optimum focal position and number of photographed images, thusmaking it possible to suppress a processing amount, a memory capacity,and power consumption to a minimum.

(Third Embodiment)

An imaging device of a third embodiment of the invention has a similarconfiguration to that of the imaging device 1 illustrated in FIG. 1.Though it is described in the embodiments above that the setting valueof the lens driving amount and characteristics of the distance at whichfocus is achieved change in accordance with the orientation of theimaging device, the characteristics may change, for example, when theimaging device is subjected to a strong impact or by aging. Thus, in theimaging device of the present embodiment, by estimating again thesetting value of the driving amount and the characteristics of thedistance, at which focus is achieved, by way of calibration, influenceof the change in the characteristics is reduced.

For example, an example when the characteristics change due to a strongimpact such as a fall will be described with reference to FIG. 10. InFIG. 10, a horizontal axis indicates the setting value of the lensdriving amount and a vertical axis indicates the distance from theimaging device to an object. Each curve of FIG. 10 illustrates thedistance at which the lens is driven with a certain setting value of thedriving amount and focus is achieved at a focal position at that time,and illustrates characteristics of a center value of the in-focus range.

When a relation between the setting value of the lens driving amount inmanufacturing and the distance becomes as indicated in FIG. 10(a) andthe characteristics which change due to a strong impact such as a fallbecomes as indicated in FIG. 10(b), the lens driving amounts with whichfocus is achieved at distances of 50 cm and 100 cm are respectively g1and g2 in FIG. 10(a) in manufacturing and g1′ and g2′ in FIG. 10(b)after the impact is given. However, the characteristics of FIG. 10(b)are unclear when the imaging device is actually used. Thus, thecharacteristics after being changed are able to be estimated bydisposing objects at distances of 50 cm and 100 cm from the imagingdevice and measuring the lens driving amounts by focusing on therespective objects with the AF function. In a case where it is assumedthat a shape of the curve indicating the characteristics does not changelike the characteristics of FIG. 10, when the lens driving amounts withwhich focus is achieved at any several distances are found as describedabove, it is possible to estimate the entire characteristics as FIG.10(b). It is only required that the estimated characteristics are newlystored in the information storage unit 12 and the setting value of thelens driving amount is set based on the information.

However, it is actually supposed that the shape of each of the curves ofFIG. 10 changes, for example, the characteristics change more greatly inthe near view side or the distant view side. In this case, by disposingobjects at all distances from the near view to the distant view forphotographing and focusing on each of them with the AF to read the lensdriving amount, the changed characteristics are able to be known, but isit not realistic in consideration of labor of photographing and aprocessing amount. Thus, objects at several distances of, for example,50 cm, 100 cm, 300 cm and the like are photographed and the lens drivingamount by the AF are read, and new characteristics is estimated so thatit is not deviated from the shape of the curve of the characteristics inmanufacturing. As the number of distances at which photographing isperformed increases, reliability of characteristics to be estimatedbecomes high, so that it is desirable that the distances at whichphotographing is performed are set by calibration in consideration oflabor of processing with the increase in the number of photographedimages and estimation accuracy for the characteristics.

Note that, the calibration is able to be performed by causing the userto perform photographing by giving an instruction such as “pleasephotograph an object at a distance of 50 cm”. However, it may be saidthat disposing an object at a specified distance and performingphotographing a plurality of times give a great burden to the user.Thus, an object which has a determined size as a standard, for example,such as a B-4 sheet or a newspaper, is photographed, and a distance tothe object is calculated by an actual object size, a pixel size on animage, and optical characteristics such as a focal distance. By focusingon the object with the AF and reading the lens driving amount, arelation between the distance to the object and the lens driving amountis able to be known. For example, by photographing an object whose sizeis known at some distances while moving the imaging device in front andback directions, it is possible to estimate the lens driving amount andthe characteristics of the distance at which focus is achieved in arange where the imaging device is moved, even when the object is notdisposed at a correct distance. However, even when photographing isperformed a plurality of times at almost the same distance, only a partof the characteristics is able to be estimated, so that it is desirableto perform photographing while moving the imaging device in front andback directions in a range as wide as possible because an estimationrange is expanded.

In addition, it may be configured such that by using the fact that aface of a person has almost the same size, a face of a person isdetected by a known facial detection technique or the like and broughtinto focus with the AF and a distance is obtained by a size of the faceto estimate characteristics. In this case, the calibration may beperformed automatically by using a portrait image which is photographedby the user without being conscious of the calibration.

Note that, it is also possible to improve accuracy of the calibration byphotographing an object at one distance a plurality of times andcalculating an average value and the like. Accordingly, it is desirablethat an object to be photographed, a distance to the object, the numberof photographed images and the like are set in consideration of a burdenon the user and the accuracy of the calibration.

Note that, description has been given above by assuming that informationof the setting value of the driving amount in manufacturing, which isstored in the information storage unit 12, is updated to informationestimated by the calibration and the updated setting value of thedriving amount is set. However, it may be configured so that a changingamount of the changed setting value of the driving amount is acquired bythe calibration and the setting value of the driving amount which isoriginally set is corrected with a correction amount according to thechanging amount.

With the method described above, the imaging device of the presentembodiment estimates the setting value of the lens driving amount andthe characteristics of the distance at which focus is achieved, whichchange due to an impact, aging or the like, by the calibration. Sincethe setting value of the driving amount is set based on the estimatedcharacteristics, it is possible to perform photographing at theappropriate focal position and easily acquire an image having anenlarged depth of field.

By using an object whose size is known for the calibration or using faceinformation of a person or the like, it is possible to carry out thecalibration by reducing the burden on the user.

(Fourth Embodiment)

An imaging device of a fourth embodiment of the invention includes theaperture mechanism 15 in the optical system 101 of the imaging device 1in the embodiment above, and a case where a depth of field of theimaging device is variable by a user operation is assumed.

When the aperture mechanism 15 is included, it is possible to change thedepth of field by stopping down the lens. An aperture level of the lensis represented by an F-number, and the F-number is a value obtained bydividing a focal distance by an effective aperture. When the F-numberchanges by the aperture and the depth of field changes, the settingvalue of the lens driving amount and the characteristics of the distanceas illustrated in FIG. 6 and FIG. 7 also change. Thus, by storing thesetting value of the driving amount according to the change of theaperture, that is, the change of the depth of field in the informationstorage unit 12 and reading information from the information storageunit 12 based on aperture information such as the F-number, the settingvalue of the driving amount is set. This makes it possible to set thefocal position appropriately even when the aperture changes and thedepth of field changes.

Note that, though description has been given above by assuming a casewhere the setting value of the driving amount according to apertureinformation is stored in the information storage unit 12, information ofa correction amount according to the aperture information may be storedin the information storage unit 12. Thereby, the correction amount maybe read based on the aperture information to correct the setting valueof the driving amount which is set as a reference.

With the method described above, the imaging device of the presentembodiment sets the setting value of the driving amount according to anaperture level of the lens. Thus, also when the imaging device includingthe aperture mechanism is used, it is possible to set the focal positionappropriately in accordance with the depth of field which changes due tothe aperture, perform image processing for a photographed image, andappropriately acquire an image having an enlarged depth of field.

(Fifth Embodiment)

It is described in the embodiments above that even when the orientationof the imaging device changes, an image having an enlarged depth offield is able to be acquired by appropriately setting the focal positionand performing photographing. When the orientation changes excessively,however, an operation of the lens driving unit may become unstable. Forexample, when there is an excessive change of the orientation bydirecting the imaging device directly downward or directly upward, thelens is driven with a maximum or minimum driving amount in some cases.The lens driving unit easily operates unstably when the lens is drivento both end positions of a driving range because of a mechanisticproblem. Accordingly, even when the setting value of the driving amountis set based on orientation information, photographing may not beperformed at the desired focal position.

The imaging device of the fifth embodiment of the invention gives awarning or an instruction to the user in order to prevent such anorientation of the imaging device, by which the lens is driven unstably.Note that, the imaging device of the present embodiment includes adevice for giving warning/instruction, such as a lamp, or a displaydevice or a speaker, as the alarm unit 16 as illustrated in FIG. 1 inthe configuration of the imaging device 1 in the embodiments above.

For example, when an angle of the optical axis is inclined by 70 degreesor more from a horizontal state, the imaging device of the presentembodiment notifies the user, for example, by way of making a lamp forwarning included in the imaging device blink or sounding an alarm forwarning. Thereby, the excessively inclined orientation is notified, andthe user adjusts the orientation of the imaging device byhimself/herself so as to perform stable photographing. When the imagingdevice includes a display device, characters such as “please make theorientation almost horizontal” may be displayed on the display device,or when including a speaker, an instruction may be given by sound. Anylimitation to the orientation for giving a warning or an instruction,that is, any threshold for deciding the orientation may be set, and itis desirable that the threshold is set so that the setting value of thedriving amount is able to be set in a range where the lens is able to bedriven most stably.

With the method described above, the imaging device of the presentembodiment is able to prevent that the orientation of the imaging devicechanges excessively and driving of the lens becomes unstable by giving awarning or an instruction to the user in accordance with orientationinformation. Thereby, by performing photographing while driving the lensstably and changing the focal position, and performing image processingfor a photographed image, it is possible to acquire an image having anenlarged depth of field.

A part of the imaging device 1 of the embodiments described above, forexample, the image processing unit 13 may be realized by a computer. Inthis case, it may be realized by recording a program for realizing afunction of the processing unit in a computer readable recording mediumand causing a computer system to read the program recorded in therecording medium. Note that, the “computer system” described here is acomputer system embedded in the imaging device 1 and is assumed toinclude the OS and hardware of peripheral devices and the like.Moreover, the “computer readable recording medium” refers to a portablemedium such as a flexible disc, an optical magnetic disc, a ROM or aCD-ROM, or a storage device such as a hard disc embedded in a computersystem. Further, the “computer readable recording medium” includes onewhich dynamically holds a program for a short time, such as acommunication line in a case where the program is transmitted through anetwork such as the Internet or a communication line such as a telephoneline, and one which holds a program for a fixed time, such as a volatilememory inside a computer system serving as a server or a client in theabove case. The aforementioned program may be one for realizing a partof the functions described above, and further may be one capable ofrealizing the functions described above by being combined with a programwhich has been already recorded in a computer system.

A part of the imaging device 1 in the embodiments described above may berealized as an integrated circuit such as LSI (Large Scale Integration).Each functional block of the imaging device 1 may be realized asindividual processors, or a part or all thereof may be integrated into aprocessor. Furthermore, the circuit integration method is not limited tothe LSI and may also be realized with dedicated circuits or generalprocessors. Further, in a case where a technique for making into anintegrated circuit in place of the LSI appears with advance of asemiconductor technique, an integrated circuit by the technique may beused.

In the above embodiments, the configurations and others illustrated inthe accompanying drawings are not limited, but changes can be made asappropriate within the range in which the effect of the invention isexerted. Further, the invention can be implemented by modifying asappropriate as long as it does not depart from the scope of the objectof the invention. Any selection can be made optionally from eachcomponent of the invention, and an invention which includes the selectedconfiguration is also included in the invention.

For example, though processing for enlarging the depth of field has beendescribed, it is also possible to apply to processing for adjusting thedepth of field, which includes reduction.

(Appendix)

The invention includes the following disclosure.

(1) An imaging device, including:

an image acquisition unit for acquiring a plurality of images havingdifferent focal positions;

an orientation acquisition unit for acquiring orientation information ofthe image acquisition unit; and

an image processing unit for generating, from the plurality of images,an image having a depth of field which is enlarged compared to a depthof field of one of the plurality of images, in which

a focal position is determined based on a focal position setting valuewhich is corrected based on the orientation information acquired by theorientation acquisition unit.

Thereby, it is possible to set the focal position appropriately andacquire an image having an enlarged depth of field even when anorientation of the imaging device changes. Moreover, sincecharacteristics of the depth of field determined in accordance with anoptical system and a condition for photographing are considered, andfurther, the focal position is set based on in-focus information withrespect to a reference object, the reference object is always in focusand the image having the enlarged depth of field is able to be acquiredby suppressing the number of images to be photographed to a minimum.This makes it possible to suppress a processing amount, a memorycapacity, and power consumption which increase in accordance with thenumber of images.

(2) The imaging device according to (1), in which

the orientation information has information about an angle formed by anoptical axis of the image acquisition unit and a vertical direction, and

a sign of a correction amount of the focal position setting value isvaried based on a sign of the angle information.

(3) The imaging device according to (1) or (2), in which the orientationinformation has information about an angle formed by an optical axis ofthe image acquisition unit and a vertical direction, and

a correction amount of the focal position setting value is varied basedon magnitude of the angle information.

The correction amount of the setting value of the driving amount is alsoable to be varied in accordance with the magnitude of the angleinformation.

(4) The imaging device according to any one of (1) to (3), in which

the focal position setting value is determined based on one referencefocal position setting value, and

a correction amount of other focal position setting value correctedbased on the orientation information varies in accordance with thereference focal position setting value.

In this manner, the setting value of the driving amount is able to becorrected and set appropriately in accordance with the orientationinformation.

(5) The imaging device according to (4), in which based on theorientation information and the reference focal position setting value,a proportion of the number of other focal position setting values whichresult in a near view compared to an image acquired at a focal positionof the reference focal position setting value to the number of otherfocal position setting values which result in a distant view compared tothe image acquired at the reference focal position is adjusted.

This makes it possible to perform photographing at the appropriate focalposition according to the position of the reference object and toenlarge the depth of field.

(6) The imaging device according to any one of (1) to (5), furtherincluding a zoom mechanism, and

an information storage unit in which a setting value of a lens drivingamount according to presence/absence of zooming or a degree of zoomingis stored, in which

distance information of an object acquired from in-focus information andthe setting value of the driving amount according to the distanceinformation are set.

This makes it possible to acquire an image having an enlarged depth offield with the optimum focal position and number of photographed imagesaccording to zooming information and in-focus information.

(7) The imaging device according to any one of (1) to (5), in which achanged setting value of a lens driving amount and characteristics of adistance at which focus is achieved are estimated by calibration.

Since the setting value of the driving amount is set based on theestimated characteristics, it is possible to perform photographing atthe appropriate focal position and acquire an image having an enlargeddepth of field.

(8) The imaging device according to any one of (1) to (5), in which asetting value of a lens driving amount according to an aperture level ofa lens is set.

Also when the imaging device including an aperture mechanism is used, itis possible to set the focal position appropriately in accordance withthe depth of field which changes due to the aperture, perform imageprocessing for a photographed image, and acquire an image having anenlarged depth of field.

(9) The imaging device according to any one of (1) to (5), including analarm unit for giving an alarm in accordance with the orientationinformation.

By giving a warning or an instruction, it is possible to prevent thatthe orientation of the imaging device changes excessively and driving ofthe lens becomes unstable. Thereby, by performing photographing whiledriving the lens stably and changing the focal position, and performingimage processing for a photographed image, it is possible to acquire animage having an enlarged depth of field.

(10) A processing method in an imaging device including an imageacquisition unit for acquiring a plurality of images having differentfocal positions, an orientation acquisition unit for acquiringorientation information of the image acquisition unit; and an imageprocessing unit for generating, from the plurality of images, an imagehaving a depth of field which is enlarged compared to a depth of fieldof one of the plurality of images, in which

a focal position is determined based on a focal position setting valuewhich is corrected based on the orientation information acquired by theorientation acquisition unit, and

the focal position setting value is corrected based on the orientationinformation.

(11) A program for causing a computer to execute the processing methodaccording to (10).

(12) A computer readable recording medium having the program accordingto (11) recorded therein.

REFERENCE SIGNS LIST

1 imaging device

10 image acquisition unit

11 orientation acquisition unit

12 information storage unit

13 image processing unit

100 imaging element

101 optical system

102 lens driving unit

103 driving amount setting unit

2 a object in near view

2 b object in distant view

a1, b1 to b3, c0 to c2, d0 to d2, e0 to e2, f0 to f2, g1, g2, g1′, g2′setting value of driving amount

ar, br, cr0 to cr2 in-focus range

A, B, C, D curve

All publications, patents and patent applications cited in thisspecification are incorporated herein by reference in their entirety.

The invention claimed is:
 1. An imaging device, comprising: an imageacquisition unit for acquiring a plurality of images having differentfocal positions; an orientation acquisition unit for acquiringorientation information of the image acquisition unit; and an imageprocessing unit for generating, from the plurality of images, an imagehaving an in-focus range wider than an in-focus range of each of theplurality of images, wherein a focal position setting value by which afocal position is determined is provided, and the focal position settingvalue is corrected based on the orientation information.
 2. The imagingdevice according to claim 1, wherein the orientation information hasinformation about an angle formed by an optical axis of the imageacquisition unit and a vertical direction, and a sign of a correctionamount with which the focal position setting value is corrected isvaried based on a sign of the angle information.
 3. The imaging deviceaccording to claim 1, wherein the orientation information hasinformation about an angle formed by an optical axis of the imageacquisition unit and a vertical direction, and a correction amount withwhich the focal position setting value is corrected is varied based onmagnitude of the angle information.
 4. The imaging device according toclaim 1, wherein the focal position setting value is determined based ona reference focal position setting value with which a reference objectserving as a reference for determining the focal position is in focus,and a correction amount when the focal position setting value iscorrected based on the orientation information varies in accordance withthe reference focal position setting value.
 5. The imaging deviceaccording to claim 4, wherein based on the orientation information andthe reference focal position setting value, a proportion of the numberof focal position setting values with which an object closer to a nearview than the reference object is in focus to the number of focalposition setting values with which an object closer to a distant viewthan the reference object is in focus is adjusted, and based on thefocal position setting value which is adjusted, the image acquisitionunit acquires the plurality of images.
 6. The imaging device accordingto claim 1, further comprising: a zoom mechanism capable of zooming onan object to be photographed, wherein the focal position setting valueis set in accordance with presence/absence of zooming in the zoommechanism or a degree of zooming.
 7. The imaging device according toclaim 6, wherein the focal position setting value is set in accordancewith information about a distance of the object, which is decided fromin-focus information when the object is in focus.
 8. The imaging deviceaccording to claim 1, further comprising: an aperture mechanism capableof changing a depth of field, wherein the focal position setting valueis set in accordance with an aperture level of a lens in the aperturemechanism.
 9. The imaging device according to claim 1, whereincharacteristic information indicating a relation between a setting valueof a driving amount by which a lens driving amount is set and a distanceat which focus is achieved is estimated by calibration, and the focalposition setting value is set based on the estimated characteristicinformation.
 10. The imaging device according to claim 1, wherein thefocal position setting value is the setting value of the driving amountby which the lens driving amount is set.
 11. An imaging device,comprising: an image acquisition unit for acquiring a plurality ofimages having different focal positions; and an image processing unitfor generating, from the plurality of images, an image having anin-focus range wider than an in-focus range of each of the plurality ofimages, wherein in a first in-focus image and a second in-focus imagewhich are generated from an image acquired in a state where respectiveangles formed by an optical axis direction of the image acquisition unitand a vertical direction are different from each other, an in-focusdistance at which focus is achieved in the first in-focus image and anin-focus distance at which focus is achieved in the second in-focusimage are the same.
 12. The imaging device according to claim 11,wherein the image acquisition unit includes an optical system forcondensing light and an optical driving unit for controlling a positionof the optical system, and the optical driving unit changes a drivingamount with which the position of the optical system is controlled inaccordance with the angle formed by the optical axis of the imageacquisition unit and the vertical direction.